Acquisition of software application performance data is an important but difficult task because efforts to gather the performance data may change the behavior of the monitored application, which, in turn, also affects the acquired performance data. In a possible worst case, performance monitoring may cause a malfunction or even a crash of the application.
A variety of tools exist for monitoring performance by instrumenting the bytecode of the monitored application. Such tools can provide performance data at a fine-grained level (e.g., down to data describing single method calls). Information at this detailed level is powerful for quickly and efficiently finding and eliminating performance problems. Acquisition of such high-quality information may require the placement of a large quantity of bytecode for performance monitoring into the bytecode of the monitored application. Such bytecode which is inserted into the monitored application is referred to herein as “sensor bytecode” or simply as “sensors.” Sensors can create a large amount of measurement data that has to be aggregated and analyzed.
Further, monitoring of distributed applications can require the transfer of both original bytecode and instrumented bytecode over network connections. Although this process may be aided and enhanced by compression and/or encryption of the bytecode to make more efficient use of network bandwidth, network transmission of bytecode is subject to bottlenecks, both caused by network connections and the instrumentation process. For example, an unreliable or failed network connection can stall the instrumentation process. Further, in the case of distributed applications, the instrumentation process itself is subject to bottlenecks at the multiple, parallel instrumentation engines which are used in different processes on different computers.
Information about application performance data is often created by the applications that are monitored and then transferred to remote monitoring systems which may correlate performance data received from multiple applications. Propagation of application performance data from creating applications to a monitoring system is subject to different fluctuating latencies introduced by the behavior of the computer networks interconnecting monitored applications with monitoring systems. Additionally, the timing systems maintained by the computer systems running the monitored applications and the timing systems of the compute systems running the monitoring system may be subject to different time drifts.